Information displaying apparatus

ABSTRACT

An information displaying apparatus includes: a memory control unit that stores information to a storage medium, and outputs the information from the storage medium; a display that displays the information output from the storage medium in a predetermined display form on a screen of the display; an image-capturing unit that captures an image of an observer in a state to observe the display and outputs an image signal; a facial image detection unit that detects an inclination of a facial image of the observer based on the image signal; and a display form change unit that changes the display form of the information to be displayed on the screen. The display form change unit changes the display form so that the information to be displayed on the screen rotates by a predetermined rotation angle based on the inclination of the facial image detected by the facial image detection unit.

INCORPORATION BY REFERENCE

The disclosure of the following priority application(s) is/are hereinincorporated by reference:

Japanese Patent Application No. 2008-002902 filed Jan. 10, 2008.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information displaying apparatusthat controls information to be displayed based on a state of the faceof a viewer who has been photographed.

2. Description of Related Art

A display with good visibility to the viewer can be done by displayingstill images information in close-up by using the technology disclosedin Japanese Laid-Open Patent Application No. 2005-26933. JapaneseLaid-Open Patent Application No. 2007-264124 discloses among others atechnology that performs image display that is easily viewablecorresponding to a difference in eye color and to a surrounding colortemperature. With the technology disclosed in Japanese Laid-Open PatentApplication No. 2007-67782, it is possible to specify persons by theirfacial images taken and control functions of equipment available to therespective persons.

SUMMARY OF THE INVENTION

However, with the technology disclosed in Japanese Laid-Open PatentApplication No. 2005-26933, it is necessary for the viewer to perform anoperation in one way or another in order to achieve a display which iseasy to see; for example, the viewer has to perform an operation tospecify the degree of expansion when it is intended to display in anexpanded fashion. Therefore, it takes time for the user to go to thetrouble to specify the degree of the expansion when the distance betweenthe display screen and the user is changed to make the expansionunnecessary any more, or when he or she wants to increase the degree ofthe expansion.

With the technology disclosed in Japanese Laid-Open Patent ApplicationNo. 2007-264124, only the brightness of the display is appropriatelyadjusted. Therefore, no display that is easy to see can always beobtained depending on the positions of the displaying apparatus and theviewer. Use of the technology disclosed in Japanese Laid-Open PatentApplication No. 2007-67782 makes it possible to set the display screenso as to meet the preference of each user. However, there is noguarantee that the display is always easy to see when the positions ofthe displaying apparatus and the viewer are changed.

In view of the above-mentioned problems, the present invention is madeto obtain a display that is easy to be seen by a viewer by an automaticoperation to control the form of display information based on a facialimage of the viewer (observer) who looks into the display screen.

An information displaying apparatus according to a 1st aspect of thepresent invention comprises: a memory control unit that storesinformation to a storage medium, and outputs the information from thestorage medium; a display that displays the information output from thestorage medium in a predetermined display form on a screen of thedisplay; an image-capturing unit that captures an image of an observerin a state to observe the display and outputs an image signal; a facialimage detection unit that detects an inclination of a facial image ofthe observer based on the image signal; and a display form change unitthat changes the display form of the information to be displayed on thescreen, and in the information displaying apparatus the display formchange unit changes the display form so that the information to bedisplayed on the screen rotates by a predetermined rotation angle basedon the inclination of the facial image detected by the facial imagedetection unit.

According to the 2nd aspect of the present invention, in the informationdisplaying apparatus according to the 1st aspect, it is preferred thatthe information comprises index information that visually conveys anevent and a content to the observer.

According to the 3rd aspect of the present invention, in the informationdisplaying apparatus according to the 2nd aspect, the index informationmay comprise at least one of a character, a sign, a figure, and an icon.

According to the 4th aspect of the present invention, in the informationdisplaying apparatus according to the 1st aspect, the information maycomprise image information.

According to 5th aspect of the present invention, in the informationdisplaying apparatus according to the 1st aspect, it is preferred thatthe display form change unit rotates the information based on theinclination of the facial image so that the information is substantiallyin front of and upright to the observer to change the display form.

According to the 6th aspect of the present invention, it is preferredthat the information displaying apparatus according to the 1st aspectfurther comprises: a directional information determination unit thatdetermines whether the information includes directional information, andin the information apparatus, the display form change unit may determinethe rotation angle based on the inclination of the facial image and thedirectional information when it is determined by the directionalinformation determination unit that the information includes thedirection information.

According to the 7th aspect of the present invention, in the informationdisplaying apparatus according to the 6th aspect, the display formchange unit may determine a standard display size of the informationbased on the inclination of the facial image and the directionalinformation.

According to the 8th aspect of the invention, it is preferred that theinformation displaying apparatus according to the 1st aspect furthercomprises: a directional information determination unit that determineswhether the information includes directional information, and in theinformation displaying apparatus, the facial image detection unitfurther includes: a pattern determination unit that determines whetherthere is a specific pattern in the information. In the informationdisplaying apparatus, when it is determined by the directionalinformation determination unit that no directional information isincluded in the information and when it is determined by the patterndetermination unit that the specific pattern is included in theinformation, the display form change unit may determine directionalinformation based on the specific pattern.

According to the 9th aspect of the present invention, in the informationdisplaying apparatus according to the 1st, it is preferred that theimage detection unit further comprises a pattern determination unit thatdetermines whether there is a specific pattern in the information, andthe information displaying apparatus further comprises: a directionalinformation determination unit that determines whether the informationincludes directional information; and a prohibition unit that prohibitsrotation of the information by the display form change unit when it isdetermined by the directional information determination unit that nodirectional information is included in the information and when it isdetermined by the pattern determination unit that no specific pattern isincluded in the information.

According to the 10th aspect of the present invention, the informationdisplaying apparatus according to the 1st aspect may further comprise: amode switch unit that switches operation between a replay mode andanother mode other than the replay mode; and a recognition unit thatrecognizes that there is a plurality of the facial images detected bythe facial image detection unit, and in the information displayingapparatus, if it is recognized by the recognition unit that there is aplurality of the facial images while the mode other than the replay modeis being executed, the mode switch unit may switch the mode from themode other than the replay mode to the replay mode.

According to the 11th aspect of the present invention, in theinformation displaying apparatus according to the 1st aspect, thedisplay form change unit may comprise a size change unit that changes asize of display of the information based on a size of the facial imagedetected by the facial image detection unit.

According to the 12th aspect of the present invention, in theinformation displaying apparatus according to the 11th aspect, it ispreferred that the size change unit performs expansion or reductionprocessing so that the smaller the size of the facial image, the largerthe information is displayed.

According to the 13th aspect of the present invention, in theinformation displaying apparatus according to the 11th aspect, it ispreferred that the size change unit comprises at least one of: a displaysize selection unit that selects a predetermined size of display from aplurality of sizes of display included in the information based on thefacial image; and an expansion/reduction unit that expands/reduces,respectively, the information at a predetermined magnification dependingon the size of the facial image.

According to the 14th aspect of the present invention, the informationdisplaying apparatus according to the 13th aspect may further comprise:a size determination unit that determines a size of the facial imagedetected by the facial image detection unit, and in the informationdisplaying apparatus, it is preferred that when it is determined by thesize determination unit that the size of the facial image is not lessthan a predetermined first threshold, the expansion/reduction unitexpands or reduces the information at a predetermined magnificationbased on the first threshold, and when it is determined by the sizedetermination unit that the size of the facial image is not more than apredetermined second threshold, the expansion/reduction unit expands orreduces the information at a predetermined magnification based on thesecond threshold.

According to the 15th aspect of the present invention, the informationdisplaying apparatus according to the 1st aspect may further comprise: arecognition unit that recognizes that there is a plurality of the facialimages detected by the facial image detection unit, and in theinformation displaying apparatus, it is preferred that when a pluralityof the facial images is recognized by the recognition unit, the displayform change unit changes the display form based on a facial imageclosest to a predetermined point in an image-capturing area captured bythe image-capturing unit.

According to the 16th aspect of the present invention, in theinformation displaying apparatus according to the 1st aspect, it ispreferred that the display is adapted to display the information in aplurality of directions in different display forms, and when the facialimage detection unit detects a plurality of the facial images in aplurality of predetermined areas in an image-capturing area captured bythe image-capturing unit, the display form change unit changes thedisplay form based on a facial image in each area.

According to the 17th aspect of the invention, the informationdisplaying apparatus according to the 1st aspect may further comprise: apower OFF unit that turns OFF power supply to a portion other than thefacial image detection unit of the information displaying apparatus whenno facial image is detected by the facial image detection unit for notless than a predetermined time.

According to the 18th aspect of the present invention, the informationdisplaying apparatus according to the 17th aspect may further comprise:a power ON unit that turns ON power supply when a facial image isdetected within a predetermined time from a time at which the powersupply to the portion other than the facial image detection unit of theinformation displaying apparatus has been turned OFF by the power OFFunit.

An information displaying apparatus according to the 19th aspect of thepresent invention comprises: a memory control unit that storesinformation to a storage medium and outputs the information from thestorage medium; a display that displays the information output from thestorage medium in a predetermined display form on a screen of thedisplay; an image-capturing unit that captures an image of an observerin a state to observe the display and outputs an image signal; a facialimage detection unit that detects a position at which a facial image ofthe observer occupies the image-capturing area based on the imagesignal; and a display form change unit that changes the display form ofthe information to be displayed on the screen. In the informationdisplaying apparatus, the display form change unit changes the displayform so that the information to be displayed on the screen is distortedbased on the position at which the facial image occupies theimage-capturing area detected by the facial image detection unit.

According to the 20th aspect of the invention, in the informationdisplaying apparatus according to the 19th aspect, it is preferred thatthe information comprises index information that visually conveys anevent and a content to the observer.

According to the 21st aspect of the present invention, in theinformation displaying apparatus according to the 20th aspect, it ispreferred that the index information comprises at least one of acharacter, a sign, a figure, and an icon.

According to the 22nd aspect of the present invention, in theinformation displaying apparatus according to the 19th aspect, it ispreferred that the information comprises image information.

According to the 23rd aspect of the present invention, the informationdisplaying apparatus according to the 19th aspect may further comprise:a mode switch unit that switches operation between a replay mode andanother mode other than the replay mode; and a recognition unit thatrecognizes that there is a plurality of the facial images detected bythe facial image detection unit, and in the information displayingapparatus, it is preferred that if it is recognized by the recognitionunit that there is a plurality of the facial images while the mode otherthan the replay mode is being executed, the mode switch unit switchesthe mode from the mode other than the replay mode to the replay mode.

According to the 24th aspect of the present invention, in theinformation displaying apparatus according to the 19th aspect, it ispreferred that the display form change unit includes a size change unitthat changes a size of display of the information based on a size of thefacial image detected by the facial image detection unit.

According to the 25th aspect of the present invention, in theinformation displaying apparatus according to the 24th, it is preferredthat the size change unit performs expansion or reduction processing sothat the smaller the size of the facial image, the larger theinformation is displayed.

According to the 26th aspect of the present invention, in theinformation displaying apparatus according to the 24th aspect, it ispreferred that the size change unit comprises at least one of: a displaysize selection unit that selects a predetermined size of display from aplurality of sizes of display included in the information based on thefacial image, and an expansion/reduction unit that expands/reduces,respectively, the information at a predetermined magnification dependingon the size of the facial image.

According to the 27th aspect of the present invention, the informationdisplaying apparatus according to the 26th may further comprise: a sizedetermination unit that determines a size of the facial image detectedby the facial image detection unit, and in the information displayingapparatus, it is preferred that when it is determined that the size ofthe facial image is not less than a predetermined first threshold, theexpansion/reduction unit expands or reduces the information at apredetermined magnification based on the first threshold, and when it isdetermined that the size of the facial image is not more than apredetermined second threshold, the expansion/reduction unit expands orreduces the information at a predetermined magnification based on thesecond threshold.

According to the 28th aspect of the present invention, in theinformation displaying apparatus according to the 19th aspect, it ispreferred that the display form change unit elongates the information ifthe position at which the facial image occupies the image-capturing areais outside a predetermined area.

According to the 29th aspect if the present invention, in theinformation displaying apparatus according to the 28th aspect, it ispreferred that the display form change unit elongates the informationdisplayed on the screen in an area remoter from the observer at a higherelongation ratio than the information displayed on the screen in an areacloser to the observer.

According to the 30th aspect of the present invention, the informationdisplaying apparatus according to the 19th aspect may further comprise:a recognition unit that recognizes that there is a plurality of thefacial images detected by the facial image detection unit, and in theinformation displaying apparatus, it is preferred that when a pluralityof the facial images has been recognized by the recognition unit, thedisplay form change unit changes the display form based on a facialimage closest to a predetermined point in the image-capturing areacaptured by the image-capturing unit.

According to the 31st aspect of the present invention, in theinformation displaying apparatus according to the 19th aspect, it ispreferred that the display is adapted to be capable of displaying theinformation in a plurality of directions in different display forms, andwhen the facial image detection unit detects a plurality of the facialimages in a plurality of predetermined areas in the image-capturing areacaptured by the image-capturing unit, the display form change unitchanges the display form based on a facial image in each area.

According to the 32nd aspect of the present invention, the informationdisplaying apparatus according to the 19th aspect may further comprise:a power OFF unit that turns OFF power supply to a portion other than thefacial image detection unit of the information displaying apparatus whenno facial image is detected by the facial image detection unit for notless than a predetermined time.

According to the 33rd aspect of the present invention, the informationdisplaying apparatus according to the 32nd aspect may further comprise:a power ON unit that turns ON power supply when a facial image isdetected within a predetermined time from a time at which the powersupply to the portion other than the facial image detection unit of theinformation displaying apparatus is turned OFF by the power OFF unit.

An information displaying apparatus according to the 34th aspect of thepresent invention comprises: a memory control unit that stores firstinformation and second information to a storage medium and outputs thefirst information and the second information from the storage medium; adisplay that displays the first information and the second informationoutput from the storage medium in predetermined display forms on ascreen of the display; an image-capturing unit that captures an image ofan observer in a state to observe the display and outputs an imagesignal; a facial image detection unit that detects an inclination of afacial image of the observer based on the image signal; and a displayform change unit that changes the display form of at least one of thefirst information and the second information to be displayed on thescreen, and in the information displaying apparatus the display formchange unit changes the display form so that at least one of the firstinformation and the second information to be displayed on the screenrotates at a predetermined rotation angle based on the inclination ofthe facial image detected by the facial image detection unit.

An information displaying apparatus according to the 35th aspect of thepresent invention comprises: a memory control unit that stores firstinformation and second information to a storage medium and outputs thefirst information and the second information from the storage medium; adisplay that displays the first information and the second informationoutput from the storage medium in predetermined display forms on ascreen of the display; an image-capturing unit that captures an image ofan observer in a state to observe the display and outputs an imagesignal; a facial image detection unit that detects a position at which afacial image of the observer occupies the image-capturing area based onthe image signal; and a display form change unit that changes thedisplay form of at least one of the first information and the secondinformation to be displayed on the screen, and in the informationdisplaying apparatus the display form change unit changes the displayform so that at least one of the first information and the secondinformation to be displayed on the screen is distorted based on theposition at which the facial image occupies the image-capturing areadetected by the facial image detection unit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective front view showing the construction of a digitalcamera according to the present invention;

FIG. 2 is a perspective rear view showing the construction of thedigital camera shown in FIG. 1;

FIG. 3 is a view showing the internal construction of the digital camerashown in FIGS. 1 and 2;

FIG. 4 is a diagram showing the construction of the digital camera shownin FIGS. 1 and 2 seen from above and an image-capturing area;

FIG. 5 is a diagram showing the internal construction of a buffermemory;

FIG. 6 is a diagram showing each side of image information;

FIG. 7 is a flowchart that illustrates processing for assisting displayexecuted in the replay mode;

FIG. 8 is a flowchart that illustrates the processing for assistingdisplay executed in the replay mode and in a menu mode.

FIG. 9 is a flowchart that illustrates processing for recognizing aface;

FIG. 10 is a flowchart that illustrates processing of index information;

FIG. 11A illustrates an example of the inclination of a facial image ofa viewer (observer) and FIG. 11B is a diagram showing the relationshipbetween a face direction vector and a reference vector;

FIG. 12A illustrates an example of the inclination of a facial image ofa viewer (observer) and FIG. 12B is a diagram showing the relationshipbetween a face direction vector and a reference vector;

FIG. 13A illustrates an example of the inclination of a facial image ofa viewer (observer) and FIG. 13B is a diagram showing the relationshipbetween a face direction vector and a reference vector;

FIG. 14A illustrates an example of the inclination of a facial image ofa viewer (observer) and FIG. 14B is a diagram showing the relationshipbetween a face direction vector and a reference vector.

FIG. 15 is a diagram that illustrates an example of the inclination ofdisplayed index information;

FIG. 16 is a diagram that illustrates an example of the inclination ofdisplayed index information;

FIG. 17 is a diagram that illustrates an example of the inclination ofdisplayed index information;

FIG. 18 is a diagram that illustrates an example of the inclination ofdisplayed index information;

FIG. 19 is a diagram showing an example of the size of a facial image ofthe viewer;

FIG. 20 is a diagram showing an example of the size of a facial image ofthe viewer;

FIG. 21 is a diagram showing an example of the size of a facial image ofthe viewer;

FIG. 22 is a diagram that illustrates an example of the size ofdisplayed index information;

FIG. 23 is a diagram that illustrates an example of the size ofdisplayed index information;

FIG. 24 is a diagram that illustrates an example of the size ofdisplayed index information;

FIG. 25 is a flowchart that illustrates the image informationprocessing;

FIG. 26 is a diagram showing an example of a rotation angle β determinedby a combination of the inclination φ of the facial image and thedirection of the image;

FIG. 27 is a diagram illustrating processing that determines directioninformation on an image based on the image information and the facialimage in the image information;

FIG. 28 is a diagram that illustrates an example of a display in whichthe image information shown in FIG. 27 is displayed as being rotated;

FIG. 29 is a diagram that illustrates an example of a display in whichthe image information shown in FIG. 27 is displayed as rotated;

FIG. 30 is a diagram that illustrates an example of a display in whichthe image information shown in FIG. 27 is displayed as being rotated;

FIG. 31 is a diagram that illustrates an example of the size of adisplay of image information;

FIG. 32 is diagram that shows an example of the size of displayed imageinformation;

FIG. 33 is a diagram that illustrates an example of the size ofdisplayed image information;

FIG. 34 is a flowchart that illustrates processing in which a pluralityof faces is recognized;

FIG. 35 is a diagram showing a position at which the facial image iscaptured;

FIG. 36 is a diagram showing a position at which the facial image iscaptured;

FIG. 37 is a diagram showing a position at which the facial image iscaptured;

FIG. 38 is a diagram that illustrates the form of a display of indexinformation;

FIG. 39 is a diagram that illustrates an example of the inclination of adisplay of index information;

FIG. 40 is a diagram that illustrates the form of displayed indexinformation;

FIG. 41 is a flowchart of the processing in the replay mode;

FIG. 42 is a flowchart that illustrates the elongation processing on theindex information;

FIG. 43 is a diagram that illustrates an example of the displayingdirection of an LCD that is capable of displaying in three directions;

FIG. 44 is a diagram showing a position at which the facial image iscaptured; and

FIG. 45 is a diagram that illustrates an example in which differentimages are displayed on an LCD that is capable of displaying images inthree directions.

DESCRIPTION OF PREFERRED EMBODIMENTS

A digital camera equipped with the information displaying apparatusaccording to the present invention is illustrated and an embodimentthereof is described below. In the embodiment detailed below, thepresent invention is applied to a digital camera. However, the presentinvention can be embodied in other forms as far as it relates toelectronic equipment that includes a display device. The digital cameraaccording to an embodiment of the invention will now be described by wayof example with reference to FIGS. 1 through 45.

First Embodiment

1. Arrangement

FIGS. 1 and 2 each are a perspective view showing a digital cameraaccording to the present invention. A power supply button 100 mounted onan upper part of the digital camera shown in FIG. 1 is a button thatinstructs turning ON or OFF of the power supply of the digital camera.On a surface 101 of the digital camera, there is provided a photographiclens 102 that takes optical information of a subject in order to capturean image of the subject. A release button 103 is mounted on the upperpart of the digital camera. It is a button that instructs start ofimage-capturing using the photographic lens 102. A surface 104 shown inFIG. 2 is usually a user side (viewer side and observer side) of thedigital camera. On the surface 104, there are provided an LCD 105, aphotographic lens 106, an operation unit 107 for performing varioussettings therewith, a replay button 108, a shooting mode button 109, anda menu button 110. The LCD 105 is a liquid crystal panel that displaysthe image information, icons, characters, figures, and signs, and so on

A side 111 is an upper edge of the LCD 105, a side 112 is a left edge ofthe LCD 105, a side 113 is a lower edge of the LCD 105, and a side 114is a right edge of the LCD 105. The photographic lens 106 is a lens usedto capture an image of a user (viewer or observer) of the digitalcamera. The operation unit 107 is an operation member used to move acursor displayed in the LCD 105 or make a decision on a selectedfunction. The digital camera of this embodiment has three modes, i.e., areplay mode in which the image stored in a recording medium is replayed,a shooting mode in which an image of a subject is captured, and a menumode in which various settings of the camera are made. Theabove-mentioned various modes are adapted to be switchable among them bypressing the replay button 108, the shooting mode button 109, and themenu button 110, respectively.

The internal construction of the digital camera according to anembodiment of the present invention will now be described with referenceto the block diagram shown in FIG. 3. The inside structure of thedigital camera includes an image sensor 201, an image sensor 202, an A/Dconversion circuit 203 (analogue/digital conversion circuit), an A/Dconversion circuit 213, a bus 204, a buffer memory 205, an I/O unit 206,a recording medium 207, an image processing unit 208, a flash RAM 209, aCPU 210, a display control unit 211, and a posture detection unit 212.The A/D conversion circuit 203, the buffer memory 205, the I/O unit 206,the image processing unit 208, the flash RAM 209, the CPU 210, and thedisplay control unit 211 are connected to each other through the bus204.

The image sensor 201 and the image sensor 202 are photoelectric devices,which are elements to convert an optical image into an analog electricalsignal. The image sensor 201 is an image sensor provided in order tocapture an image of a subject in the shooting mode and is disposed in alater stage of the photographic lens 102. The image sensor 202, which isto capture an image of the user (viewer) of the digital camera, isdisposed in a later stage of the photographic lens 106. The photographiclens 106 in the first embodiment is described as including a lenscapable of capturing an image of a wide angle area of about 20 mm interms of a focal length of 35 mm size film camera (having a field angleθ nearly equal to 94 (θ≈94)) as shown in FIG. 4.

The A/D conversion unit 203 converts analog electric signals output fromthe image sensor 201 into digital image signals. Similarly, the A/Dconversion unit 213 converts analog electric signals output from theimage sensor 202 into digital image signals. The image informationdigitalized by the A/D conversion circuit 203 is once accumulated in thebuffer memory 205 through the bus 204. The buffer memory 205 isconstituted by a VRAM area 205 a where the image information displayedin the LCD 105 is temporarily accumulated, an OSD area 205 b toaccumulate index information displayed in the LCD 105 temporarily, andanother area 205 c as shown in FIG. 5. The image information digitalizedby the A/D conversion circuit 213 is once stored in the other area 205 cthrough the bus 204.

The recording medium 207 is a detachable recording medium that preservesthe image information, and it is connected with the bus 204 through theI/O unit 206. Saving of the image information into the recording medium207 and reading out the image information from the recording medium 207are performed based on the instruction from the CPU 210. The imageprocessing unit 208 is a unit that performs detection of facial imagesof the viewers, determination of the number of facial images of theviewers, determination of the size of the facial image of the viewers,determination of the inclination of the facial images of the viewers,and determination of positions of the facial images of the viewers. Italso performs determination as to whether there is a facial image in theimage information and determination of the inclination of the facialimage in the displayed image information as well as processing of theimage information. The image processing unit 208 is driven according tothe instruction from the CPU 210. The CPU 210 controls the power supplyof the digital camera.

The posture detection unit 212 detects the posture of the digital camera(top and bottom, and right and left). There have been known a method ofdetecting a posture in which the electroconductivity of the pendulum asdisclosed in Japanese Laid-open Patent Application No. 2005-236883 isused to detect the posture and a method in which the metal sphere asdisclosed in Japanese Laid-open Patent Application No. 2004-286614 isused to detect the posture. The posture detection unit 212 may beadapted to use these methods to determine the posture of the camerabased on two axes, one being the direction of the gravity accelerationand the other being vertical to the direction of the gravityacceleration and parallel to the surface 101 of the camera.Alternatively, it may use a known technology using a gyro, etc.(Japanese Laid-open Patent Application No. 2005-260666). The CPU 210imparts the image information acquired by capturing an image via thephotographic lens 102 with the directional information (top and bottom,and right and left) based on the posture detected by the posturedetection unit.

Now, an On Screen Display (hereafter, OSD) is described. The display onthe LCD 105 is executed by using the OSD function. This function is todisplay an icon, a character, a figure, and a sign, etc. (indexinformation) over a replayed picture displayed in the replay mode or ona through image displayed in the shooting mode of the display in theplay mode in superposition. In this embodiment, the index information tobe displayed is stored to the flash RAM 209. The CPU 210 storespredetermined index information in the OSD area 205 b according to thestatus in which the camera is used by the user in stages before it isdisplayed in the LCD 105. The image processing unit 208 performsprocessing for changing the form of display based on the state of facialimage of the user and stores the index information in the OSD area 205 bagain. The image information to be displayed is stored in the buffermemory 205 by the CPU 210 in stages before it is displayed in the LCD105. The display control unit 211 controls the image information and theindex information such that the index information stored in the OSD areais displayed on the image information in superposition on the LCD 105.The details of the processing by the camera are described later on.

The display assistance function is described. The display assistancefunction is concerned with the image information and index informationdisplayed on the LCD 105. This function is to display information thathas undergone expansion/reduction processing and rotation processingbased on a relative position of the digital camera and the viewer unlikean ordinary display mode. The visibility to the viewer can be improvedby using this function. The display assistance is described separatelyfor the display assistance in the replay mode and the display assistancein the shooting mode/menu mode as follows. The case where no displayassistance is used is described with reference to FIG. 6. It is assumedthat the LCD 105 displays image information with which directioninformation is imparted such that the side 115 represents an upper edgeportion, the side 116 represents a left edge portion, the side 117represents a lower edge portion, and the side 118 represents a rightedge portion, respectively, of the image as shown in FIG. 6. In thiscase, the side 115 is on the side 111, the side 116 is on the side 112,the side 117 is on the side 113, and the side 118 is on the side 114.Also, the image is displayed such that the length of the side 115 isequal to that of the side 111, the length of the side 116 is equal tothat of the side 112, the length of the side 117 is equal to that of theside 113, and the length of the side 118 is equal to that of the side114.

2. Display Assistance in the Play Mode

The processing of display assistance in the replay mode is now explainedwith reference to the flowchart presented in FIG. 7. In step S1, theimage processing unit 208 performs processing for recognizing a facialimage of the viewer who observes the LCD 105 or who is located in aplace where it is possible to observe the LCD 105 (hereafter, thisprocessing is referred to as “face recognition processing”). In step S2,the image processing unit 208 performs processing the index information(hereafter, “index information processing”) according to the instructionfrom the CPU 210 and the control proceeds to step S3. In step S3, theimage processing unit 208 processes the index information according tothe instruction from the CPU 210 and the control proceeds to step S4. Instep S4, the display control unit 211 controls the image information andthe index information such that the index information temporarilyaccumulated in the OSD area 205 b is displayed in superposition over theimage information temporarily stored in the VRAM 205 a.

In step S5, the image processing unit 208 determines whether the facialimage of the viewer is recognized. If no facial image is recognized, thecontrol shifts to step S1. If a facial image is recognized, the controlproceeds to step S6. In step S6, if it is determined by the imageprocessing unit 208 that the size of facial image or the inclination inthe facial image of the viewer of an image-capturing plane has changedto a predetermined extent or more, the control shifts to step S1. If ithas been determined that the change has occurred to an extent less thanthe predetermined extent, the control shifts to step S7. In step S7, theCPU 210 determines whether end of the replay mode (shift to other modesor turning OFF the power) has been instructed by the user. The controlproceeds to step S8 when the end of the replay mode has not beeninstructed. In step S8, the CPU 210 determines whether the indexinformation has been changed (for instance, the icon has been changeddue to a decrease in residual quantity of the battery). The controlshifts to step S1 when it is determined that the index information hasbeen changed. The control shifts to step S9 when it is determined thatthe index information has not been changed. In step S9, the CPU 210determines whether the replay image has been changed. The control shiftsto step S1 when it is determined that the replay image has been changed.The control shifts to step S5 when it is determined that the replayedimage has not been changed. In step S7, the processing is ended when itis determined that it has been instructed by the user to end the replaymode. The recognition processing S1 on the facial image mentioned above,the processing S2 on the index information, and the processing of theimage information will be detailed later separately.

3. The Display Assistance in the Shooting Mode/Menu Mode

The processing of display assist in the shooting mode and in the menumode is explained in reference to the flowchart presented in FIG. 8. Inthe display assistance in the shooting mode and in the menu modeaccording to this embodiment, only the form of index information isprocessed. In the shooting mode, the processed index information isdisplayed on a through image of the background in superposition. In themenu mode, a through image, a replay image or a wallpaper, etc. aredisplayed in superposition over the index information as a backgroundimage. The shooting mode and the menu mode are usually used by one user.Therefore, when plural facial images are recognized in the shooting modeand in the menu mode, a message to invite the user to switch the mode tothe replay mode is displayed. In step S11, the image processing unit 208performs recognition of a plurality of facial images. In step S12, theimage processing unit 208 performs recognition of a facial image. Instep S13, the image processing unit 208 processes the index informationand the control proceeds to step S14. In step S14, the display controlunit 211 controls the image information and the index information suchthat the index information temporarily accumulated in the OSD area 205 bis displayed in superposition over the image information as a backgroundtemporarily accumulated in the VRAM 205 a.

In step S15, the image processing unit 208 determines whether the facialimage of the viewer has been recognized. If no facial image has beenrecognized, the control shifts to step S12 and if a facial image hasbeen recognized, the control proceeds to step S16. In step S16, theimage processing unit 208 determines whether there is only one facialimage of the viewer. The control proceeds to step S17 when only onefacial image of the viewer has been recognized, and when plural facialimages of the viewer have been recognized, the control shifts to stepS11. In step S17, if it is determined by the image processing unit 208that the size of facial image or the inclination in an image-capturingplane of facial image of the viewer has changed to at least apredetermined extent, the control shifts to step S12. If the change hasoccurred to an extent below the predetermined extent, the control shiftsto step S18. In step S18, the CPU 210 determines whether end of the modehas been instructed by the user. The control proceeds to step S19 whenthe end of the replay mode has not been instructed. In step S19, the CPU210 determines whether the index information has been changed. When itis determined that the index information has been changed, the controlshifts to step S12 and the index information is processed. If it isdetermined in step S18 that end of the mode has been instructed by theuser, this processing is ended. The recognition processing S11 on aplurality of facial images mentioned above, the recognition processingS12 on a facial image, and the processing of the index information willbe detailed later separately.

4. Recognition Processing of a Facial Image

The recognition processing on a facial image is described in referenceto the flowchart presented in FIG. 9. In step S101, if it is determinedby the image processing unit 208 that a facial image of the viewer hasbeen recognized, the control shifts to step S104, and if no facial ofthe viewer has been recognized, the control shifts to step S102. In stepS102, the CPU 210 starts counting elapsed time T and the control shiftsto step S103. In step S103, CPU 210 determines whether the elapsed timeT exceeded a predetermined elapsed time T1. T1 is assumed to be oneminute in this embodiment. The control proceeds to S105 if the elapsedtime T is less than one minute, and the control proceeds to step S106 ifthe elapsed time T is one minute or more.

In step S105, if it is determined by the image processing unit 208 thata facial image of the viewer has been recognized or the CPU 210 hasdetected an operation by the user, the control shifts to step S104. Onthe other hand, in step S105, if it is determined by the imageprocessing unit 208 that no facial image of the viewer has beenrecognized and the CPU 210 has detected no operation by the user, thecontrol shifts to step S103. In step S106, the display control unit 211displays on the LCD 105 a message to ask if it is acceptable to switchthe mode to an energy-saving mode.

In this embodiment, “ENERGY-SAVING MODE, YES or NO?” is displayed toinvite the user to decide by selecting either one with touching a cursorkey. In step S107, the CPU 210 determines whether the selection ofenergy-saving mode has been made by the user. The control proceeds tostep S108 if it is determined by the CPU 210 that the energy-saving modehas been selected by the user, and the control proceeds to step S113 ifthe energy-saving mode has not been selected. In step S108, the CPU 210determines whether YES or NO has been selected. The control proceeds tostep S109 if it is determined that YES has been selected, and thecontrol proceeds to step S113 if NO has been selected. In step S109, thepower supply to parts excluding the mechanism necessary for the facerecognition by the digital camera (for instance, power supply etc. tothe LCD 105) is turned OFF as instructed by the CPU 210 to switch theoperation mode to the energy-saving mode.

In step S110, the CPU 210 temporarily stores in the OSD area 205 bpredetermined index information to be displayed and the control proceedsto step S111. In step S111, the CPU 210 determines whether the presentmode is the replay mode. If it is the replay mode, the control proceedsto step S112, where the image information to be replayed is temporarilyaccumulated in the VRAM 205 a, and the control shifts to step S4 shownin the flowchart of FIG. 7. On the other hand, when it is determined instep S111 that the current mode is other than the replay mode, thecontrol shifts to step S14 shown in the flowchart of FIG. 8.

In step S113, the CPU 210 determines whether the elapsed time T hasreached T2 or more. T2 is assumed to be two minutes in this embodiment.Therefore when the elapsed time T2 is 2 minutes or more, the controlproceeds to step S109 and the CPU 210 executes the energy-saving mode.On the other hand, when the elapsed time T2 is less than 2 minutes, thecontrol shifts to step S107.

In step S109, if the energy-saving mode is executed, the controlproceeds to step S114. If the image processing unit 208 recognizes afacial image or if the CPU 210 determines that there has been anoperation of the digital camera by the user, the control proceeds tostep S115. On the other hand, in step S114, the control proceeds to stepS116 when the image processing unit 208 recognizes no facial image andthe CPU 210 determines that there has been no operation of the digitalcamera by the user.

In step S115, the CPU 210 turns ON the power supply made to turn OFFpartially in step S109 to cancel the energy-saving mode, and the controlproceeds to step S104. In step S116, the CPU 210 determines whether theelapsed time T has reached or exceeds T3. T3 is assumed to be fiveminutes in this embodiment. Therefore, when the elapsed time T is 5minutes or more, the control proceeds to step S117. On the other hand,when the elapsed time T is less than 5 minutes, the control proceeds tostep S114. In step S117, the power supply to the digital camera isturned OFF by the CPU 210. Moreover, the CPU 210 determines whetherthere has been an operation by the user in step S104. When there hasbeen no operation by the user, the control returns to the main flow toend this processing. On the other hand, if it is determined that therehas been an operation by the user, the control proceeds to step S118where the CPU 210 accepts the input operation and performs appropriateprocessing in response to the input operation. Then, the control returnsto the main flow.

5. Processing of Index Information

The processing procedure of the index information will be described withreference to the flowchart shown in FIG. 10 illustrating the processingof the index information and FIGS. 11A through 24. In step S301, the CPU210 reads in the index information saved in the flash RAM 209 and oncesaves it in the OSD area 205 b. Then, the control proceeds to step S302.In step S302, it is determined by the image processing unit 208 at whatan angle the face of the viewer is inclined with respect to the digitalcamera in a plane of the captured image. Then, the control proceeds tostep S303.

FIG. 11A-FIG. 14B show the relationships between images captured by useof the photographic lens 106 and the angles of inclination of facialimages. The area enclosed by a side 312, a side 313, a side 314, and aside 315 is an image-capturing area. The side 315 on the right of theimage-capturing area corresponds to the side 112, which is the left sideof the LCD. The side 313 on the left of the image-capturing areacorresponds to the side 114, which is the right side of the LCD. Theside 312 on the top of the image-capturing area corresponds to the side111, which is the top side of the LCD. The side 314 on the bottom of theimage-capturing area corresponds to the side 113 on the bottom of theLCD.

Accordingly, when describing hereafter based on the top and bottom, andthe right and left of the LCD, a facial image 308 a of FIG. 11A shows afacial image when the LCD is looked into from below. A facial image 309a of FIG. 12A shows a facial image when the LCD is looked into from theright. A facial image 310 a of FIG. 13A shows a facial image when theLCD is looked into from above. Facial image 311 a of FIG. 14A shows thefacial image when LCD is looked into from the left. A reference vector307 that serves as a criterion of the inclination of the facial image isa vector parallel to the side 313 and the side 315 headed from the lowerside to the upper part. Face direction vectors 308 b-311 b, which arevectors each indicating the direction of a face, indicate upwarddirections of the facial images 308 a-311 a (the mouth being lower andthe eyes being upper), respectively. The upward direction of the facialimage is detected by using a well-known image detection technology.

Single index information processing is distributed to steps S306-S309 bysteps S303-S305 based on the value of angle φ formed between thereference vector 307 and the face direction vector, and a rotation angleα is determined in steps S306-S309. The area of φ in which 315°≦φ<360°and 0°≦φ<45° is assumed to be a quadrant A, the area of φ, in which45°≦φ<135° is assumed to be a quadrant B, the area of φ in which135°≦φ<225° is assumed to be a quadrant C, and the area of the φ inwhich 225°≦φ<315° is assumed to be a quadrant D as shown in FIG. 11B,FIG. 12B, FIG. 13B, and FIG. 14B. When the face direction vector is inthe quadrant A, α=0°. When the face direction vector is in the quadrantB, α=27°. When the face direction vector is in the quadrant C, α=180°.When the face direction vector is in the quadrant D, α=90°.

Therefore, the rotation is not performed by the image processing unit208 shown in FIG. 15 when the inclination of the facial image 308 a isas shown in FIG. 11A and FIG. 11B. In the case where the inclination ofthe facial image of the viewer corresponds to the inclination of thefacial image 309 a as shown in FIG. 12A and FIG. 12B, 270° rotation isgiven clockwise by the image processing unit 208 with FIG. 15 beingassumed to be 0° (standard) as shown in FIG. 16. In the case where theinclination of the facial image of the viewer corresponds to theinclination of the facial image 310 a as shown in FIG. 13A and FIG. 13B,180° rotation is given clockwise by the image processing unit 208 withFIG. 15 being assumed to be 0° (standard) as shown in FIG. 17. In thecase where the inclination of the facial image of the viewer correspondsto the inclination of the facial image 311 a as shown in FIG. 14A andFIG. 14B, 90° rotation is given clockwise by the image processing unit208 with FIG. 15 being assumed to be 0° (standard) as shown in FIG. 18.As mentioned above, in step S310, rotation by only a determined rotationangle is given clockwise by the image processing unit 208 with FIG. 15being assumed to be 0° (standard). Rotation of the index information byonly a rotation angle α clockwise causes the inclination of the facialimage to approach an angle at which it can be viewed by the vieweralmost in front upright.

A step S311 and subsequent steps relate to display sizes of the indexinformation. In step S311, the image processing unit 208 determines asize L of the facial image of the viewer. The size of the facial imageis determined by using a well-known technology. The longer the distancebetween the digital camera and the viewer (i.e., the smaller the facialimage in the image-capturing plane), the poorer is the visibility of theimage. For this reason, the image processing unit 208 controls the sizeof the index information such that the smaller the size of thedetermined facial image, the larger the displayed index information.When it is determined by the CPU 210 that the size L of the facial imageis smaller than a predetermined threshold L1 in step S312, the controlproceeds to step S314. In step S314, the image processing unit 208 setsmagnification m to m3 and the control shifts to step S317. When it isdetermined by the CPU 210 that the size L of the facial image is L1≦L≦L2in step S313, the control proceeds to step S315. In step S315, the imageprocessing unit 208 sets magnification m to m2 and the control shifts tostep S317. When it is determined by the CPU 210 that the size L of thefacial image is not L1≦L≦L2 in step S313, the control proceeds to stepS316. In step S316, the image processing unit 208 sets magnification mto m1 and the control shifts to step S317. In this embodiment, it isassumed that m1, m2, and m3 are m1<m2<m3. In the following description,the display size when the size L of the facial image is larger than L2is assumed to be a standard size. In a step S317, the image processingunit 208 expands the index information m fold and the control proceedsto step S318.

In this embodiment, the size L of the facial image is obtained based onthe ratio of the area of facial image to the area of the captured imagearea the facial image occupies. The predetermined threshold L1 isassumed to be 4% in the facial image area, and this corresponds to adistance between the digital camera and the viewer of about 50 cm. L2 isassumed to be 10% of the facial image area, and this corresponds to adistance between the digital camera and the viewer of about 30 cm. Thedimensions of the size of the facial image of the viewer and of the sizeof the index information are described with reference to FIG. 19-FIG. 21and FIG. 22-FIG. 24. Facial images 301 a-301 c are facial images of theviewer captured through the photographic lens 106. In the case of thefacial image 301 a, the facial image area occupies about 20% of thecaptured image area and the size L of the facial image is shown largerthan the predetermined size L2. In the case of the facial image 301 c,the facial image area occupies about 2% of the captured image area andthe size L of the facial image is shown smaller than the threshold L1.In the case of the facial image 301 b, the facial image area occupiesabout 8% of the captured image area. Thus, the size L of the facialimage is L1≦L≦L2. Moreover, 302 a, 302 b, 302 c, 303 a, 303 b, 303 c,304 a, 304 b, 304 c, 305 a, 305 b, and 305 c shown in FIG. 22, FIG. 23,and FIG. 24 are pieces of index information displayed in the replaymode. 302 a, 302 b, and 302 c indicate date and time (year, month, timeand minute) of shooting. 303 a, 303 b, and 303 c indicate numbers of setpixels at the time of shooting. 304 a, 304 b, and 304 c orders of imagescurrently displayed/total number of image frames. 305 a, 305 b, and 305c indicate residual quantity of the battery.

The present invention will be explained taking an example of the indexinformation displayed in the replay mode. In the case of the facialimage 301 a shown in FIG. 19, the distance from which the viewervisually checks the LCD 105 is near enough. Therefore, the viewer canvisually check the index information displayed in a standard size like302 a-305 a shown in FIG. 22, so that the image processing unit 208 doesnot expand the index information. In other words, in this case, m1 isone fold in this embodiment. In the case of the facial image 301 b shownin FIG. 20, that is, the viewer is located at a slightly remoteposition, the image processing unit 208 processes the index informationsuch that the index information is displayed in a larger size like 302b-305 b compared with 302 a-305 a shown in FIG. 23. In this case, themagnification m is set to m2=1.5 in this embodiment and the indexinformation is displayed as expanded to 1.5 fold in length and breadth.In the case of the facial image 301 c shown in FIG. 21, that is, theviewer is located at a still remoter position, the image processing unit208 processes the index information such that the index information isdisplayed in a still larger size like 302 c-305 c shown in FIG. 24 ascompared with 302 b-305 b. In this case, the magnification m is set tom3=2 in this embodiment and the index information is displayed asexpanded to 2 fold in length and breadth. In step S318, the processedindex information is once accumulated in the OSD area 205 b by the CPU210. When plural facial images are detected, the image processing unit208 performs the above-mentioned processing based on the facial image ofthe person who is the nearest to the center point of the image-capturingarea.

6. Processing of Image Information

Subsequently, the processing of the index information will be describedwith reference to the flowchart shown in FIG. 25 and FIGS. 26 through33. Rotation processing of image information is processing in whichtaking into consideration directional information of the image, thedirection of an image is made to approach a direction along which theimage information is positioned upright in front of the facial image ofthe viewer. The directional information of the image can be imparted toa Tag information on Exchangeable image file format (hereafter, “Exif”),which is a file type of image widely used. The CPU 210 determines thedirection of display by reading out the directional information in theExif Tag in this processing. In step S401, the CPU 210 reads out theExif Tag information stored in the VRAM area 205 a and the controlproceeds to step S402. In step S402, it is determined by the CPU 210whether there is directional information in the Exif Tag in the imageinformation.

If there is directional information, the control proceeds to step S403.If there is no directional information, the control proceeds to stepS404. In step S403, rotation is performed in consideration of thedirectional information in addition to the direction of the facial imageof the viewer. Therefore, the rotation angle is determined by the CPU210 based on the combination of the quadrant in which the face directionvector is present with the image orientation. In step S403, rotationangle β is read in from the table shown in FIG. 26 and the controlproceeds to step S405. In step S405, the CPU 210 reads in the imageinformation saved in the VRAM region 205 a and the control proceeds tostep S406. In step S406, the image processing unit 208 rotates theread-in image information by only the rotation angle β clockwise and thecontrol proceeds to step S407. In step S407, the image information isonce saved by the CPU 210 in the VRAM area 205 a and the controlproceeds to step S411.

On the other hand, in the case where there is a facial image in theimage information that contains no directional information, the imageprocessing unit 208 determines the direction vector of the facial imagein the image information, and determines that the side to which thevector is directed is an upper side as shown in FIG. 27. In step S404,the image processing unit 208 determines whether there is a facial imagein the facial image. If there is a facial image in the imageinformation, the control proceeds to step S408. In step S408, the imageprocessing unit 208 determines a face direction vector from the facialimage of a person in the image information and determines directionalinformation of the image information. When there are plural facialimages, the image processing unit 208 determines the direction based onthe facial image nearest to the center point of the image-capturingarea. In step S409, the CPU 210 imparts the Exif Tag with theabove-mentioned directional information. In the case of the imageinformation shown in FIG. 27, the side 116 is determined to be the upperside. In step S409, if the directional information is imparted to theExif Tag in the image information, the control shifts to step S403. Instep S404, when it is determined by the image processing unit 208 thatthere is no facial image in the image information, the control proceedsto step S410. In step S410, the CPU 210 sets the rotation angle β to 0°and the control proceeds to step S411.

Step S411 and subsequent steps are steps of expansion processing of theimage information. In step S411, the CPU 210 reads out the imageinformation from the VRAM area 205 a and the control proceeds to stepS412. In step S412, the CPU 210 determines whether the rotation angle βis 270° or 90°. If β is 270° or 90°, the control proceeds to step S413and the image processing unit 208 processes the image information to astandard size. In the case of standard size, the shorter side of the LCD105 and the longer side of the image information have the same lengths,and the processing is given like the image shown in FIG. 28. In stepS413, the image processing unit 208 processes the image information byfixing the aspect ratio of the image information such that the longerside of the image information and the shorter side of the LCD 105 havethe same lengths. Then, the control proceeds to step S415.

On the other hand, the control proceeds to step S414 if the rotationangle β is other than 270° or 90° (i.e., in the case of β=0° or 180°),and the image processing unit 208 processes the image information to astandard size. In the case of the standard size, the longer side of theLCD 105 and the longer side of the image information have the samelengths and the image processing unit 208 processes the imageinformation to a size such that the longer side of the image informationand the longer side of the LCD 105 have the same lengths with fixing theaspect ratio in step S414. Next, in step S415, the image processing unit208 expands the image information such that it may become m fold. Forthe magnification m, the same value as that of m set in the indexinformation processing is used. The image obtained by processing theimage of FIG. 28 to 1.5 fold is displayed as shown in FIG. 29, and theimage expanded two fold is displayed as shown in FIG. 30. On the otherhand, the image obtained by processing the image of FIG. 31 to 1.5 foldis displayed as shown in FIG. 32, and the image expanded 2 fold isdisplayed as shown in FIG. 33. Subsequently, in step S416, the processedimage information is accumulated in the VRAM area 205 a again by the CPU210 and the control returns to the main loop. In the replay mode, theimage information of which replay is instructed by the user (viewer) isread out from the recording medium by the CPU 210 beforehand, and istemporarily stored to the VRAM area 205 a.

7. Plural Faces Recognition Processing

Processing for recognizing plural faces (hereafter, referred to as“plural faces recognition processing”) is now explained in reference tothe flowchart presented in FIG. 34. In step S501, it is determined bythe image processing unit 208 whether there is a plurality of facialimages of viewers captured by the image sensor 202. When there is aplurality of facial images, the control proceeds to step S502 whereaswhen there are no plural images, the control returns to the main loop.In step S502, the display control unit 211 displays a message to askwhether to shift to the replay mode in the LCD 105, and the controlproceeds to step S503. In step S503, the input from the user isaccepted. If it is “YES”, the control proceeds to step S1 shown in FIG.7 in order to shift to the replay mode. If it is “NO”, the controlreturns to the main loop.

According to the first embodiment as described above, the followingadvantages are obtained.

(1) The image processing unit 208 is adapted to recognize the facialimage of a viewer (observer) looking into the LCD 105 based on the imagesignal output from the image sensor 202, and determines a rotation angleabased on an angle of the face direction vector to the reference vector,that is, the inclination of the facial image. As a result, the indexinformation on which the rotation processing is performed by only therotation angle α is displayed, that is, the index information isdisplayed so as to confront the viewer almost in front and upright inthe LCD 105 based on the inclination of the facial image. Therefore, thevisibility of the index information by the viewer can be improved.

(2) The CPU 210 is adapted to determine whether directional informationis included in Exif Tag in the image information and determines therotation angle β based on the inclination of the facial image and thedirectional information when the directional information is included. Asa result, the image on which the rotation processing is performed byonly the rotation angle β is displayed in the LCD 105, and the verticaldirection of the LCD 105 as seen by the viewer and the verticaldirection of the displayed image information agrees with each other.Therefore, the visibility of the image information to the viewer can beimproved.

(3) The image processing unit 208 processes the image information to astandard size so that the shorter side of the LCD 105 and the longerside of the image information may have the same lengths when therotation angle β is 270° or 90°. In addition, the image processing unit208 processes the image information to a standard size so that thelonger side of the LCD 105 and the shorter side of the image informationmay have the same lengths when the rotation angle β is 0° or 180°. Thatis, since the image processing unit 208 determines a standard displaysize of the image information based on the inclination of the facialimage to the image-capturing plane and the directional information, theimage information to be easily seen by the viewer can be obtained by theautomatic operation.

(4) The image processing unit 208 is adapted to determine thedirectional information such that the orientation of direction vector ofthe facial image is found to be toward the upper side of the imageinformation when the direction vector of the facial image in the imageinformation is determined in case the image information does-not containthe directional information. Therefore, the vertical direction of theLCD 105 to the viewer and the vertical direction of the displayed imagecan be matched even if the directional information is not contained inthe image information and the image information that is easily seen bythe viewer can be obtained by the automatic operation, so thatconvenience improves.

(5) The image processing unit 208 is adapted to determine whether thereare plural facial images of viewers captured by the image sensor 202 inthe shooting mode and in the menu mode. The display control unit 211 isadapted to display a message to ask whether to switch to the replay modein the LCD 105 when it is determined by the image processing unit 208that there are plural facial images. Therefore, when plural facialimages are detected, convenience can be improved by inviting the viewerto switch the operation mode to the replay mode since it is highlypossible that the image is viewed by plural persons in the replay mode.

(6) The image processing unit 208 is adapted to determine the size L ofthe detected facial image and change the size of index information basedon the size L of the facial image. That is, the image processing unit208 is adapted to set the magnification of index information to m3 whenthe size L of the facial image is smaller than the threshold L1 and tom1 when the size L of the facial image is greater than the threshold L2.As a result, the expansion processing or the reduction processing isperformed to the index information such that the smaller the size of thefacial image, that is, the larger the distance between the viewer andthe LCD 105, the greater the index information that can be displayed.Accordingly, it is possible to make it easy to grasp the content of theindex information by increasing the visibility of the index information.

Second Embodiment

A different point of this embodiment from the first embodiment is inthat a step of the elongation of the index information is added to thelatter part of the step of the processing of the index information. Thisembodiment is practiced with a lens with which an image within the rangeof θ being about 180° as shown in FIG. 4 can be captured.

FIGS. 35, 36, and 37 show areas of which images have been captured byusing the photographic lens 106. The image-capturing area of each ofFIGS. 35, 36, and 37 is laterally divided into three sub-areas. Thesub-areas are named area i1, area i2, and area i3, respectively. Whenthe viewer who exists in area i1 or i3 looks at (observe) the displayscreen, the visibility can be improved by displaying the indexinformation horizontally elongated by the image processing unit 208. Thedisplay area of the LCD 105 is also divided into three sub-areas and thesub-areas are named area j1, area j2, and area j3, respectively, asshown in FIGS. 38, 39, and 40. The image processing unit 208 processesthe index information at different elongation ratios depending onwhether it exists in the area i1 or the area i3 so that the indexinformation remoter from the viewer can be elongated more. When theviewer is observed in i3 as shown in FIG. 35, the index informationdisplayed in j1 is displayed by the display control unit 211 as shown inFIG. 38. In this case, the image processing unit 208 performs 1.5 foldexpansion laterally in this embodiment. When the viewer is observed ini1 as shown in FIG. 36, the index information displayed in j1 isprocessed by the image processing unit 208 and displayed by the displaycontrol unit 211 as shown in FIG. 39. In this case, the image processingunit 208 performs 2 fold elongation laterally in this embodiment. Whenthe viewer is observed in i2 as shown in FIG. 37, the image processingunit 208 does not perform elongation processing, so that the indexinformation is displayed as shown in FIG. 40.

The processing of assisting display in the replay mode is now explainedwith reference to the flowchart presented in FIG. 41. In step S21, theimage processing unit 208 performs processing for recognizing the facialimage of the viewer. In step S22, the image processing unit 208processes the index information according to the instruction from theCPU 210 and the control proceeds to step S23. In step S23, the imageprocessing unit 208 performs elongation processing on the indexinformation according to the instruction from the CPU 210 and thecontrol proceeds to step S24. In step S24, the image processing unit 208processes the index information according to the instruction from theCPU 210 and the control proceeds to step S25. In step S25, the displaycontrol unit 211 controls the image information and the indexinformation such that the index information temporarily accumulated inthe OSD area 205 b is displayed in superposition over the imageinformation temporarily accumulated in the VRAM 205 a.

In step S26, image processing unit 208 determines whether the facialimage of the viewer has been recognized. If no facial image has beenrecognized, the control shifts to step S21 and if a facial image hasbeen recognized, the control proceeds to step S27. In step S27, if it isdetermined by the image processing unit 208 that the size of facialimage or the inclination in the facial image of the viewer of animage-capturing plane or the position of facial image of the viewer haschanged to at least a predetermined extent, the control shifts to stepS21. If the change is to an extent below the predetermined extent, thecontrol shifts to step S28. In step S28, the CPU 210 determines whetherend of the replay mode has been instructed by the user. The controlproceeds to step S29 when the end of the replay mode has not beeninstructed. In step S29, the CPU 210 determines whether the indexinformation is to be changed. When it is determined that the indexinformation has to be changed, the control shifts to step S12 and theindex information processing is performed. The control shifts to stepS30 when it is determined that the index information has not to bechanged.

In step S30, the CPU 210 determines whether the replayed image haschanged. When it is determined that the replay image has changed, thecontrol shifts to step S21. The control shifts to step S26 when it isdetermined that the replay image has not changed. In step S28, the CPU210 ends this processing when it is determined that end of the replaymode has been instructed by the user. Because steps other than theelongation of the index information are similar to those in the firstembodiment, the elongation processing in this embodiment is described asfollows.

8. Elongation Processing on Index Information

Details of the steps of elongation processing on the index informationare described with reference to the flowchart presented in FIG. 42. Instep S601, the image processing unit 208 determines a column “i” inwhich the facial image of the viewer exists and the process proceeds tostep S602. When it is determined by the CPU 210 that i=i2 in step S602,the image processing unit 208 does not perform the elongation processingand the control returns to the main loop. On the other hand, the controlproceeds to step S603 when it is determined by the CPU 210 that i=i2 isnot established, and the CPU 210 determines a column “j” of the positionof display of the index information based on information on position ofdisplay (hereafter referred to as “display position information”)provided for the index information beforehand. Then, the controlproceeds to step S604. In steps S604 and S605, the next step to beadvanced is assigned as follows based on the combination (i,j) of thecolumn of facial image of the viewer and the column of display position.In the case of (i,j)=(i1,j1) or (i3,j3), the control proceeds to stepS606, and the CPU 210 sets the elongation rate X to X1. In the case of(i,j)=(i1,j2) or (i3,j2), the control proceeds to step S607, and the CPU210 sets the elongation rate X to X2. In the case of (i,j)=(i1,j3) or(i3,j1), the control proceeds to step S608, and the CPU 210 theelongation rate X to sets X3. In this embodiment, it is assumed that X1is 2 fold, X2 is 1.8 fold, and X3 is 1.5 fold. In step S609, the indexinformation is read out by the CPU 210 from the OSD area 205 b and thecontrol proceeds to step S610. In step S610, the index information iselongated X fold by the image processing unit 208 and the controlproceeds to step S611. In step S611, the processed index information isonce accumulated in the OSD area 205 b by the CPU 210 and the controlreturns to the main loop.

The processing of assisting display in the replay mode has beenexplained above. However, the effect of the improvement in visibility isalso achieved in the shooting mode and the menu mode by inserting theprocessing step of elongation of the index information in the latterpart of step S13 described in the first embodiment. In this embodiment,the image processing unit 208 has elongated only the index information.However, similar processing may be performed on the image information.In addition, the image-capturing area may be divided into threesub-areas in the vertical direction although it has been divided intothree sub-areas in the lateral direction. When facial images of pluralpeople are recognized, the image processing unit 208 performs elongationprocessing taking as a reference the position of a person who is nearestto the center point of the image-capturing area. When the facial imageextends over plural sub-areas, the image processing unit 208 determinesthat the area in which the facial image exists is the sub-area thatoccupies the facial image in the largest ratio among the pluralsub-areas. Similarly, when the index information extends over pluralareas, the image processing unit 208 determines that the indexinformation exists in the sub-area that occupies the index informationin the largest ratio among the plural sub-areas.

According to the second embodiment as mentioned above, in addition tothe advantages of the first embodiment, the following advantages can beobtained.

The image processing unit 208 is adapted to recognize the facial imageof the viewer (observer) based on the image signal output from the imagesensor 202, and performs elongation processing on the index informationbased on the position that the facial image occupies in theimage-capturing area. That is, the image processing unit 208 is adaptedto perform elongation processing on the index information displayed inan area remoter from the viewer on the LCD 105 at a higher elongationratio than on the index information displayed in an area closer to theviewer on the LCD 105. As a result, the viewer can confirm the indexinformation to which the distortion is reduced regardless of theposition in which the viewer views the LCD 105, so that the visibilitycan be improved.

Third Embodiment

In the first embodiment and the second embodiment, when plural facialimages are recognized, the processing is performed based on the facialimage of a person who is closest to the center point of theimage-capturing area. On the other hand, an LCD that can displaydifferent pieces of information in three directions, respectively, isused as the LCD 105 as shown in FIG. 43 in the third embodiment.Moreover, this embodiment is executed with a lens with which an imagecan be captured within the range of θ of about 180° as shown in FIG. 4.FIG. 44 shows an area in which an image is captured by the image sensor202 by using the photographic lens 106.

The area in which an image is captured is divided into three sub-areas,i.e., area k1, area k2, and area k3, and the processing described in thefirst embodiment is performed based on the facial image of a person whois closest to the center point in each sub-area. When a single facialimage extends over plural sub-areas, the image processing unit 208determines that the area of the facial image exists in the sub-area inwhich there is a part of the facial image that occupies the facial imagein the largest ratio. As shown in FIG. 44, the image processing unit 208processes the index information and the image information based on thefacial image 401 of the person who appears in the area k1. As a result,the display control unit 211 displays the index information and theimage information in the display direction 503 on the display plane ofthe LCD 105. The image processing unit 208 processes the indexinformation and the image information based on the facial image 402 ofthe person who appears in the area k2. As a result, the display controlunit 211 displays the index information and the image information in thedisplay direction 502. The image processing unit 208 processes the indexinformation and the image information based on the facial image 403 ofthe person who appears in the area k3. As a result, the display controlunit 211 displays the index information and the image information in thedisplay direction 501. The appearance of display information displayedin each direction is as shown in FIG. 45. If there is an area where nofacial image is detected, the display control unit 211 may turn OFF thedisplay in a direction corresponding to that area. As a result, even ifplural facial images are detected in plural sub-areas in theimage-capturing area, that is, even if plural viewers look the LCD 105from plural directions, the visibility of the index information and theimage information to each viewer can be improved.

9. Supplementation

In the first embodiment, the image processing unit 208 has beenexplained to determine the direction based on the facial image when thedirectional information is imparted to the image information. However,the direction of the image may be determined based on other patterns asfar as they are specified patterns that can presume the direction of theimage. The direction may be determined based on, for instance, characterpattern or color pattern etc. of the subject, or the direction may beestimated and determined from combinations of them.

In the first to the third embodiments, the image processing unit 208determines the size of the facial image based on the ratio of the facialimage area to the image-capturing area. However, other methods may beused to obtain the size of the facial image. In addition, therelationship between the size of the facial image and the rate ofexpansion is not limited to the values shown in this embodiment.Similarly, the relationship between the position and the elongation rateof the facial image is not limited to the values shown in thisembodiment. The comparative values T1, T2, and T3 of the elapsed timeshown by the face recognition processing are not limited to the valuesindicated in this embodiment.

Moreover, although the rate of expansion of the index information or theimage information includes 3 stages, the rate of expansion may be set tobe more continuous to change it smoothly. Moreover, although the scaleof rotation angle of the index information or the image informationincludes 4 stages, the scale of rotation angle may be set to be morecontinuous to change it smoothly. Although individual rotation of theicon is shown as an example of the rotation of the index information,the layout that displays the index information may be rotated to changethe display position. The same value is used for the magnification m ofthe index information and for the magnification m of the imageinformation. However, different values may be used therefor.

In this embodiment, the expansion processing is performed on the imageinformation using the center of the image information as a referencepoint. However, the present invention is not limited thereto and thecenter of the facial image in the image may be used as a referencepoint. In the above-mentioned embodiment, a method that employsexpansion processing is used as a method of displaying the indexinformation enlarged. However, a method may also be used in which iconsand characters with different sizes are stored in the memory in advanceand a piece of index information having a predetermined size may beselected for use from the icons or characters depending on the size ofthe facial image. When plural facial images were detected in the firstembodiment and the second embodiment, the facial image of a person whois closest to the center point of the image-capturing area is assumed tobe a facial image of a criterion for the determination. However, it maybe assumed that the facial image of a viewer who is closest to aspecific point other than the center point is used as a facial image ofthe criterion. Alternatively, the facial image of the criterion may bedetermined based on the size or inclination of the facial image.

Moreover, the same is true for the detection of the facial image in theimage information and the facial image may be other than the one that isclosest to the center point of the image information. Moreover, the sameis true for the third embodiment and the facial image may be other thanthe one that is closest to the center point in each area. Moreover,although the processing is performed based on the size, inclination, andposition of a single facial image, the processing may be performed basedon an average size, an average inclination, and average position ofplural facial images.

The image-capturing area in the third embodiment is divided into threesub-areas along the direction of the longer side of the image-capturingarea. However, the image-capturing area may be divided into twosub-areas by using an LCD that can display different pieces ofinformation in two directions. Moreover, the image-capturing area may bedivided into four or more sub-areas by using an LCD that can displaydifferent pieces of information in more than four directions. Moreover,the display direction is divided in a plane vertical to the screen andparallel to the longer side of the screen. However, the displaydirection may be divided in a plane parallel to the shorter side of thescreen.

Although the present invention is adopted to the digital camera thatdisplays the image information and index information in superposition asmentioned above, the present invention may also be applied to aninformation displaying apparatus that displays only the imageinformation or only the index information. In addition, the presentinvention has been described as using the image information and theindex information. However, the present invention is not limited theretoand text information or sentence information may also be used as far assuch is information of the kind that can be displayed on a displaymedium. That is, the present invention is not limited to the digitalcameras according to the embodiments as far as the features andfunctions of the present invention are realized. For instance, thepresent invention is applicable to an information displaying apparatusequipped with a display medium constituted by a mobile phone, a PDA, aPC, an LCD such as a photo viewer or the like. In this case, theinformation displaying apparatus is assumed to be the one that includesthe photographic lens 106 and image sensor 202 for photographing atleast the viewer. The image processing unit 208 only has to performvarious types of processing on the index information and the imageinformation in the manner as mentioned above based on the inclination ofthe facial image, the size of the facial image, and the position atwhich the facial image occupies in the image-capturing area and so ondetected by using the image signal output from the image sensor 202.

The above described embodiments are examples, and various modificationscan be made without departing from the scope of the invention.

What is claimed is:
 1. An information displaying apparatus, comprising: a body; a first image-capturing unit, disposed at the body, that captures a subject image and generates first image information; a memory control unit that stores information to a storage medium and outputs the information from the storage medium, the information including the first image information generated by the first image-capturing unit and index information that visually conveys an event and a content to an observer, the index information not being captured by the first image-capturing unit; a display, disposed to face the observer at the body, that displays the information output from the storage medium in a predetermined display form on a screen of the display; a second image-capturing unit, disposed to face the observer at the body, that captures an image of the observer in a state to observe the display and outputs an image signal; a facial image detection unit that detects a relative position at which a facial image corresponding to a face of the observer occupies an image-capturing area based on the image signal output from the second image-capturing unit, the image-capturing area including the facial image of the observer not being displayed on the screen of the display; and a display form change unit that changes the display form of the information to be displayed on the screen, wherein, the display form change unit changes the display form so that the information to be displayed on the screen is distorted based on the relative position at which the facial image occupies the image-capturing area detected by the facial image detection unit.
 2. An information displaying apparatus according to claim 1, wherein, the index information comprises at least one of a character, a sign, a figure, and an icon.
 3. An information displaying apparatus according to claim 1, further comprising: a mode switch unit that switches operation between a replay mode and another mode other than the replay mode; and a recognition unit that recognizes that there is a plurality of the facial images detected by the facial image detection unit, wherein if it is recognized by the recognition unit that there is a plurality of the facial images while the mode other than the replay mode is being executed, the mode switch unit switches the mode from the mode other than the replay mode to the replay mode.
 4. An information displaying apparatus according to claim 1, wherein, the display form change unit includes a size change unit that changes a size of display of the information based on a size of the facial image detected by the facial image detection unit.
 5. An information displaying apparatus according to claim 4, wherein, the size change unit performs expansion or reduction processing so that the smaller the size of the facial image, the larger the information is displayed.
 6. An information displaying apparatus according to claim 4, wherein the size change unit comprises at least one of: a display size selection unit that selects a predetermined size of display from a plurality of sizes of display included in the information based on the facial image, and an expansion/reduction unit that expands/reduces, respectively, the information at a predetermined magnification depending on the size of the facial image.
 7. An information displaying apparatus according to claim 6, further comprising: a size determination unit that determines a size of the facial image detected by the facial image detection unit, wherein, when it is determined that the size of the facial image is not less than a predetermined first threshold, the expansion/reduction unit expands or reduces the information at a predetermined magnification based on the first threshold, and when it is determined that the size of the facial image is not more than a predetermined second threshold, the expansion/reduction unit expands or reduces the information at a predetermined magnification based on the second threshold.
 8. An information displaying apparatus according to claim 1, wherein, the display form change unit elongates the information if the position at which the facial image occupies the image-capturing area is outside a predetermined area.
 9. An information displaying apparatus according to claim 8, wherein, the display form change unit elongates the information displayed on the screen in an area remoter from the observer at a higher elongation ratio than the information displayed on the screen in an area closer to the observer.
 10. An information displaying apparatus according to claim 1, further comprising: a recognition unit that recognizes that there is a plurality of the facial images detected by the facial image detection unit, wherein, when a plurality of the facial images has been recognized by the recognition unit, the display form change unit changes the display form based on a facial image closest to a predetermined point in the image-capturing area captured by the image-capturing unit.
 11. An information displaying apparatus according to claim 1, wherein, the display is adapted to be capable of displaying the information in a plurality of directions in different display forms, and when the facial image detection unit detects a plurality of the facial images in a plurality of predetermined areas in the image-capturing area captured by the image-capturing unit, the display form change unit changes the display form based on a facial image in each area.
 12. An information displaying apparatus according to claim 1, further comprising: a power OFF unit that turns OFF power supply to a portion other than the facial image detection unit of the information displaying apparatus when no facial image is detected by the facial image detection unit for not less than a predetermined time.
 13. An information displaying apparatus according to claim 12, further comprising: a power ON unit that turns ON power supply when a facial image is detected within a predetermined time from a time at which the power supply to the portion other than the facial image detection unit of the information displaying apparatus is turned OFF by the power OFF unit.
 14. An information displaying apparatus, comprising: a body; a first image-capturing unit, disposed at the body, that captures a subject image and generates first image information; a memory control unit that stores first information and second information to a storage medium and outputs the first information and the second information from the storage medium, the first information including the first image information generated by the first image-capturing unit and the second information including index information that visually conveys an event and a content to an observer, the index information not being captured by the first image-capturing unit; a display, disposed to face the observer at the body, that displays the first information and the second information output from the storage medium in predetermined display forms on a screen of the display; a second image-capturing unit, disposed to face the observer at the body, that captures an image of the observer in a state to observe the display and outputs an image signal; a facial image detection unit that detects a relative position at which a facial image corresponding to a face of the observer occupies an image-capturing area based on the image signal output from the second image-capturing unit, the image-capturing area including the facial image of the observer not being displayed on the screen of the display; and a display form change unit that changes the display form of at least one of the first information and the second information to be displayed on the screen, wherein, the display form change unit changes the display form so that at least one of the first information and the second information to be displayed on the screen is distorted based on the relative position at which the facial image occupies the image-capturing area detected by the facial image detection unit.
 15. An information displaying apparatus, comprising: a body; a first image-capturing unit, disposed at the body, that captures a subject image and generates first image information; a memory control unit that stores the first information and second information to a storage medium and outputs the first information and the second information from the storage medium, the first information including the first image information generated by the first image-capturing unit and the second information including index information that visually conveys an event and a content to an observer, the index information not being captured by the first image-capturing unit; a display, disposed to face the observer at the body, that displays the first information and the second information output from the storage medium in predetermined display forms on a screen of the display; a second image-capturing unit, disposed to face the observer at the body, that captures an image of the observer in a state to observe the display and outputs an image signal while the first and the second information displayed on the screen of the display; a facial image detection unit that detects a relative position at which a facial image corresponding to a face of the observer occupies an image-capturing area based on the image signal output from the second image-capturing unit, the image-capturing area including the facial image of the observer not being displayed on the screen of the display; and a display form change unit that changes the display form of at least one of the first information and the second information to be displayed on the screen, wherein, the display form change unit changes the display form so that at least one of the first information and the second information to be displayed on the screen is distorted based on the relative position at which the facial image occupies the image-capturing area detected by the facial image detection unit. 